Problem Note 54366: The SAS® Web Application Server stops responding or errors occur when you restart the server
 |  |  |  |
When you restart the SAS Web Application Server, errors can occur or the server might stop responding during deployment of web applications.
When an error occurs, the server.log file for the web application server might contain messages similar to the following:
2014-10-06 11:13:25,643 [Thread-174] WARN [unknownunknown] com.sas.svcs.cluster.httpinvoker.AutoDiscoveredHttpService - Have received 1440 failures while trying to access an auto-discoverable service to satisfy request for 'http://hostname:8080/SASWIPServices/remote/urlGenerator'. Please ensure that the application and/or process surfacing this interface is still in the process of starting.
Exception in thread "Thread-174" com.sas.svcs.cluster.httpinvoker.UnavailableServiceException: The system attempted to find and access a service for interface 'com.sas.framework.config.UrlGeneratorInterface' for 2 hours and was unsuccessful.
at com.sas.svcs.cluster.httpinvoker.AutoDiscoveredHttpService.handleTimePeriods(AutoDiscoveredHttpService.java:435)
at com.sas.svcs.cluster.httpinvoker.AutoDiscoveredHttpService.invoke(AutoDiscoveredHttpService.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
For the other symptom (the server.log files stops logging during deployment of web applications), you might notice that the server never starts completely.
In this situation, the server.log file contains messages similar to the following:
2014-06-19 23:05:37,543 INFO (localhost-startStop-1) [org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/FolderModule]] Initializing Spring root WebApplicationContext 2014-06-19 23:06:06,107 INFO (localhost-startStop-2) [org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/SASAdmin]] Initializing WebApplicationContext for Struts ActionServlet 'action', module '' 2014-06-19 23:06:06,602 INFO (localhost-startStop-2) [org.apache.catalina.startup.HostConfig] Deploying configuration descriptor E:\Apps\SAS\Config\Lev1\Web\WebAppServer\SASServer1_1\conf\Catalina\localhost\SASAuthorizationServices.xml 2014-06-19 23:06:14,425 INFO (localhost-startStop-2) [org.apache.catalina.startup.HostConfig] Deploying configuration descriptor E:\Apps\SAS\Config\Lev1\Web\WebAppServer\SASServer1_1\conf\Catalina\localhost\SASBIDashboard.xml 2014-06-19 23:07:22,085 INFO (localhost-startStop-2) [org.apache.catalina.startup.HostConfig] Deploying configuration descriptor E:\Apps\SAS\Config\Lev1\Web\WebAppServer\SASServer1_1\conf\Catalina\localhost\SASBIDashboardEventGen.xml 2014-06-19 23:07:46,993 INFO (localhost-startStop-2) [org.apache.catalina.startup.HostConfig] Deploying configuration descriptor E:\Apps\SAS\Config\Lev1\Web\WebAppServer\SASServer1_1\conf\Catalina\localhost\SASBIPortlets.xml 2014-06-19 23:08:35,035 INFO (localhost-startStop-2) [org.apache.catalina.startup.HostConfig] Deploying configuration descriptor E:\Apps\SAS\Config\Lev1\Web\WebAppServer\SASServer1_1\conf\Catalina\localhost\SASBIWS.xml 2014-06-19 23:09:04,896 INFO (localhost-startStop-2) [org.apache.catalina.startup.HostConfig] Deploying configuration descriptor E:\Apps\SAS\Config\Lev1\Web\WebAppServer\SASServer1_1\conf\Catalina\localhost\SASContentServer.xml 2014-06-19 23:09:14,959 INFO (localhost-startStop-2) [org.apache.catalina.startup.HostConfig] Deploying configuration descriptor E:\Apps\SAS\Config\Lev1\Web\WebAppServer\SASServer1_1\conf\Catalina\localhost\SASDeploymentBackup.xml 2014-06-19 23:12:38,954 INFO (localhost-startStop-2) [org.apache.catalina.startup.HostConfig] Deploying configuration descriptor E:\Apps\SAS\Config\Lev1\Web\WebAppServer\SASServer1_1\conf\Catalina\localhost\SASEnterpriseMinerJWS.xml 2014-06-19 23:12:41,366 INFO (localhost-startStop-2) [org.apache.catalina.startup.HostConfig] Deploying configuration descriptor E:\Apps\SAS\Config\Lev1\Web\WebAppServer\SASServer1_1\conf\Catalina\localhost\SASEnterpriseMinerServices.xml 2014-06-19 23:13:10,581 INFO (localhost-startStop-2) [org.apache.catalina.startup.HostConfig] Deploying configuration descriptor E:\Apps\SAS\Config\Lev1\Web\WebAppServer\SASServer1_1\conf\Catalina\localhost\SASEnvironmentMgrMidTier.xml
In general, when the server start-up is complete, you should see a message similar to the following:
In this message, xxxxxxxx specifies a number.
However, in cases where the server stops responding, this message never appears in the server.log file.
One possible reason for this problem is that the Apache ActiveMQ journal logs is full. You can verify that as the source of the problem with either of the following options:
Option 1: Check the size of the folder SAS-configuration-directory/Levn/Web/activemq/data. If this folder is more than 20 GB in size, that could be the source of the problem.
Option 2: Look for the following content in the activemq log file that resides in SAS-configuration-directory\Lev1\Web\activemq\data:
2014-07-24 08:43:43,685 | INFO | Usage(default:store:queue://sas.jms.EVQueue:store) percentUsage=0%, usage=21480863010, limit=21474836480, percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=100%, usage=21480863010, limit=21474836480, percentUsageMinDelta=1%: Persistent store is Full, 100% of 21474836480. Stopping producer (ID:WX0149-59253-1406108187924-1:3:1:1) to prevent flooding queue://sas.jms.EVQueue. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 75996s) | org.apache.activemq.broker.region.Queue | ActiveMQ Transport: tcp:///11.34.80.87:3456@8676
If you see the message highlighted above (Persistent Store is Full), then that is the cause of the problem. .
In these cases, you can resolve the problem by following the steps in SAS KB0036267, "Increasing numbers of Apache ActiveMQ journal log files in certain SAS® 9.4 Enterprise BI and SAS® Visual Analytics environments fill up disk space."
Note: The increase in journal log files might also manifest in other symptoms (for example, an inability to save or open reports in SAS® Visual Analytics). This issue exists only in the SAS 9.4 TS1M0 and TS1M1.
Operating System and Release Information
| Product Family | Product | System | Product Release | SAS Release | ||
| Reported | Fixed* | Reported | Fixed* | |||
| SAS System | SAS Web Application Server | Solaris for x64 | 9.4 | 9.4 | 9.4 TS1M0 | 9.4 TS1M2 |
| Linux for x64 | 9.4 | 9.4 | 9.4 TS1M0 | 9.4 TS1M2 | ||
| HP-UX IPF | 9.4 | 9.4 | 9.4 TS1M0 | 9.4 TS1M2 | ||
| 64-bit Enabled Solaris | 9.4 | 9.4 | 9.4 TS1M0 | 9.4 TS1M2 | ||
| 64-bit Enabled AIX | 9.4 | 9.4 | 9.4 TS1M0 | 9.4 TS1M2 | ||
| Microsoft® Windows® for x64 | 9.4 | 9.4 | 9.4 TS1M0 | 9.4 TS1M2 | ||